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This paper is intended to address itself to a seemingly straight- 

X 

forward issue--namely, "Can the predictions of academic success currently 
achieved with national colleges adtnission testing programs be improved?" 

This question, however, is deceptively simple; and it begs for ampli¬ 
fication and elaboration. Indeed, it seems to us that the question 
cannot be addressed meaningfully without a consideration of at least, 

several issues pertinent to the validities of college admission testing 

) 

programs, when data elements in those programs are used tb predict 
academic attainment. In this presentation, therefore, we consider the 
prediction question in light of several general issues including: (1) 
objectives and purposes of prediction, (2) predictor and criteria consid¬ 
erations, (.3) utility and'predict ion ^nd (4) other considerations. 

Although these aspects of the prediction question are by no means pertinent 
only to testing related to college admissions concerns, this paper will r 
concentrate solely on testing in that context. 


Objective and Purposes of Predictors 


The immediate challenge facing those who would seek to improve 

the criterion>related validities of college admissions tests, where 

* 

*Thi* paper was presented at a symposium entitled "Can Predictions of ; 
Academic Success Be Improved?" at the Annual Meeting of the National 

Council on Measurement in Education, San Francisco, California , April - 

r' The jpurpos e of the paper « a * t o ^timulat e discussion of several 
l* ill ns III l„h :tj> topic. Consequently, those issues and their f 
dttfMjwp^piences are not Fully developed in the paper. ■/ 
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’’academic success” is the criterion, is to further define the objective(s) 
of prediction. One might address the question "Can predictions of 
students academic success in college be improved?” simply by investigating 
approaches to increasing the magnitude of a correlation coefficient 
relating test score performance to overall college grade-point average 
(a common approach in many prediction studies). Such an approach might 
be reasonable, but it is all too often rather simplistic, shortsighted, 
and misleading. 

Although college admission tests are popularly viewed as having'as^ 
their primary (if not single) purpose the prediction of student performance 
and the consequent selection of students, such views are not only greatly, 
exaggerated, they are rather inaccurate. Even in those instances when 
predictive data based on tests are used in the selection process, test 
scores are seldom, if. ever, used as the sole criterion for selection. 

Also, college admissions tests are used for a variety of purposes, many 
of which involve a prediction function and some of which do not. . By way 
of illustration of the predictive uses, several examples of such uses 
are cited below following a general description of a college admissions 

r v -' 

program, the ACT Assessment Program. 

The ACT Assessment Program is a multi-purpose assessment battery 
containing four ability tests, an interest inventory, and an approximately 
200-ite« Student Profile Section aimed, in part, at describing prospective 

college students and^Wfir needs, independent of test scores. The four 

...... 

tests: Engl isfcrvs age. Math Usage-, Social Studies Reading and Natural 
Sciences Reading, are each highly curriculum-related. The Program is 

TV _ - _ ■ ____ 

su pported with- array of r^rhmnal and interpretive documents and 
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services including various research services used to assist colleges and 
students in predicting the academic success of students prior to enroll- • 
ment in college. 

One goal of college admissions testing programs is to"~provide 

* i 

colleges with information on the performances of present students which 

* 

they can generalize to future students. Because < colleges ) differ on many 
important characteristics, as do the groups of students who attend those 
colleges, the acquisition and use of such information is of necessity, 
an individualized matter for the college. Accordingly, ACT provides 
Research Services that enable individual , colleges to consider the relation¬ 
ships between ACT test scores along with other pertinent data (e.g., 
high school grades and other test .scores^ of the college's choice) and 

academic performance in college. Each yeai^, ACT aids approximately 

\ 

1?500 colleges in conducting regression studies aimed at validating each 

college's particular interpretations and uses of the ACT.data. For 

« 

v * > « 

example, in a typical prediction system the ACT tests are used jointly 
with high school grades (and in some instances with othef data provided 
by colleges) to predict academic success in a number of contexts including 
estimating students': / 

(1) first-year (or first-term) college grade-point average 

(2) chances of obtaining an overall grade-point average of C or 

higher at colleges specified bjr the stjid^nts • * 

(3) chances of obtaining an av6rage > first-year grade-point average _ 
of C or higher in particular schools or programs within a 
specific college^or university 


rx 
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The above should serve to illustrate the need to view attempts to 

improve the prediction of academic success in terms of the variety of 

\ ( 

predictive Uses commonly employed. Indeed, efforts to improve predictions 

of academic performance may meet with various degrees of success as a 

\ 

function of the particular predictive use being made of the test scores 
and other data in any given circumstance. Even if such consideration is 
given, it is perhaps pertinent to observe that college admissions testing 

f 

\ 

programs'are often used for purposes other than prediction, purposes 
which cannot be ignored as researchers attempt to identify procedures 

for improving predictions. 

, « 

In addition to considering the particular uses of tests and other 

data for-predicting students' academic performance, the test developer 
* 

♦ 

must also be concerned with the various types of colleges and groups of 
students to be tested and the validities of the tests and other data for 
predictions relevant tc these colleges and groups. This is especially 

e 

true of college admissions tests which are used with diverse types of 
colleges add equally diverse groups of students. It is possible, for 

example, that interventions aimed at improving the validity of the tests 

« ^ , • 
for predicting students' academic success could achieve the desired end 

W one group, but have a negative impact on another. 

\ 

N A : particular use of the ACT tests illustrates this observation. 

Many of the colleges using the ACT tests to develop.prediction, equations 

find that th,e regression analyses conducted show that the Natural Science 

* , * , • « 
Reading Test usually does not add appreciably to the strength of the 
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would be to reduce the effectiveness of the battery for the large number 
of nursing and engineering schools who find that this test is a major 

i 

element in their prediction systems. Moreover, it migBt = °weTl be argued 
th,at, even if the ACT Natural Sciences Reading Tests failed to contribute 
in a significant way to predicting students' academic success at any 
college, the test should still be included if only because the battery 
was designed to assess knowledge and skills representative of the major 

secondary curriculum areas. To put it another way, predictive validity 

\ 

for a specific purpose is one, but only one, type of relationship in a 
network that addresses the construct validity of an instrument or program. 

A corollary observation is apropos here. It must be noted that the 

/ 

objective of a person engaged in prediction is to investigate an inter¬ 
pretation of data arising from a specified measurement procedure. 

Moreover, as Cronbach has aptly stated, "Empirical validation is inevitably 

*■ 

retrospective yet the use of the test is always prospective." 1 For 

i 

example, frequently a prediction system is developed using a group of 
examinees in a particular year, but decisions are ultimately based upon 
a different group of examinees tested in a subsequent year. Also, 

V 

> 

prediction of academic success using the data elements in college admissions 
testing programs, and efforts to sustain those improvements over time, 
are sometimes constrained by the fact that the predictor instruments/.procedures 
are in a state of flux. Among othe^things, for example, test content 
in the ACT Assessment Program, is effected periodically by evolving 

l * 

curricula. 


^Cronbach, L_.- J. "Test Validation. IhR. L. Thorndike (Ed.), 
EducationaTmeasurement (2nd ed.). Washington, D.C.: American Council * 
bn Education, 1971, p. 485. 


.Moreover, the nature, of the decisions made Based on the available data 
(and the prediction systems that.use those data) also change. Again, by 
way of example, ACT test data are clearly used much less frequently 
today in selection decisions than they were 15 or 20 years ago. Moreover, 
the data are u ;ed much more extensively for placement decisions than 
ever before, partly^because of increased heterogeneity in the abilities 
of enrolled college students. 

For the reasons cited above--multipurpose uses of test scores and 
other pertinent data involving prediction and the diversity of the 
groups which are the object of those predictions—the question "Can 
predictions of academic success in Golle l ge be improved?" has no single 
simple answer that is entirely satisfactory or useful to those who 
routinely engage in such predictions. 

Predictor and Criteria Considerations 

In considering any single predictive validity related to a college 

admissions testing program, we must, of necessity, pay careful attention 

to the predictor(s), the criterion, and their relationship(s). A thorough 

treatment of these matters is beyond the scope of this presentation, but 

we will briefly consider some issues we judge important that are perhaps * 

♦ » • 

t*oo frequently overlooked. * 

» f 

* 

First, if we assume, as is usually the case, that the academic 

success criterion is college grade-point average, then we must also 

* - 

recognize that the nature o/ both the predictor variable(s) and the cri- 

i * 

• * 

terion variable(s) impact the rel at ionship b et ween the two. _Moreover,- 

tegt developers and researchers cannot easily impact some factors (e.g., 

\ " ' ' . !' ; . . ' 7 ' ; 

practices related to grading) in the Same tanner that they can effect 




improvements in test development procedures and techniques for analyzing 
the data resulting from the use of those procedures. The problems faced 
by many of the professions who use admissions tests rn selecting first-year 
students for the professional schools are illustrative of the previous 
point. 

The applicants to a typical professional school far exceed in 

number the available spares. Moreover, these applicants are, as a 

/ 

group, usually extremely capable as demonstrated by excellent pre¬ 
professional school academic records, and equally high test scores on 
admissions examinations. The consequent restricted ranges of the pre¬ 
dictor variable(s) alone impact the effectiveness of efforts to predict 
academic success in the professional schools. Although it is theoret¬ 
ically possible to extend the ranges of these variables, it is not 
always appropriate to do so. For example, although the developer of the 
admissions tests might construct those tests to spread students out 
across a test score stale more than the current tests do^ it is likely 
that'^his could be accomplished only by drawing finer distinctions than 
are meaningful or appropriate given the goal of prediction--to select 


students who are capable of academic success in the .professional school. 

Also, it is frequently the case that the range of the success 
criterion is similarly constrained." Because the students admitted to a 
professional school typically have exceptional academic credentials, 
they usually perform extremely well s £n school. Indeed the attrition 


fate of many of the professional schools is very low, and seldom does 


attrition occur for solely academic reasons. A result is that the 

„ ' » ••• ... - •’ 

criterion for academic success is typically quite restricted inrange. 

v- . . ..•-•••. • •; .....: . . . vm... : »• 

These facts,, in part, no doubt explain the modest predictive relationships 
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so frequently obtained when undergraduate grade-pbi^^ averages are 
combined with admissions test scores to predict professional school 
grade-point averages. 

9 A second observation seems pertinent as one considers the objective . 

of improving predictions of academic success in the context of admissions 

tests'". Such tests are typically used on a continuing basis, year alter 

year. . In such cases, and when the tests (and other data) are used by 

schools that are highly selective of their students (an ever-diminishing 

number ip these days of decline in enrollment), one might expect a- 

decline in the magnitude of some types of predictive relationships of 

* 

test scores (and other data) and academic success (college GPA), if the ' 

% * 

predictive system does indeed identify students who are likely to achieve 
academic success once admitted to college. It is perhaps interesting to 
note that data on a random sample of colleges who have been selective of 
students, and who use the ACT tests along with high school grades for 
predicting academic success, indicate no such discernible declines in | 
multiple r's over the past several years. This may simply mean that the 

colleges are not relying solely on test scores (and other academic data 

\ y 

related to prior performances of students) in selecting their students. 

To the extent that this is the cas^, and to the extent that other "hidden!’ 

. | ' \ 

variables are an integral part of the admissions decisions process, 

i ' 

efforts tp increase multiple r’s (elven for selective colleges) may have 
relatively little practical significance even if they are successful. 

A cons ideration of the, ^suc-eess" criterion would not be complete if 

•. v 

one failed to call attention to the Shortcomings of using first-year 
college GPA as the sole criterion of prediction. Although it may'be 
argued that aj£pdenic success in the first year of college or a professional 









9 



school is highly relevant to subsequent academic success in college 
(since most attrition due to academic failures occurs in the first year 

r 

of college or professional school), many also argue that grade-point-average, 
per se, is not a worthy success criterion. In that connection, many 
studies have been conducted to ascertain the relationship |>etw^en achievement 
in college (and of test scores) and successful performance on the job. 

Most such studies have failed to reveal such relationships. One possible^ 
explanation for these disappointing results is that the predictor variables 
were too gross; that is, encompassed too many irrelevant and confounding 
factors. 


Some recent attempts to relate new types of measures of academic 

performance in college to success in job performance have yielded some 

j ... • 

encouraging but very mbdest results. Notable among these is the College 
Outcome Measures Project (COMP) sponsored by ACT and funded, in large 


-pa rt, by the Fuad -for /the improvement -ofPos tsecondary Education^—Thi-s- 

project, which focuses on the assessment of generic skills that rep*j£sent, 
by the concensus of the colleges participating in the project, many of 
the intended outcomes of their general education or liberal arts programs, 

is in ijts third year of' research and development. Preliminary studies ' 

/ 

suggest relationships between^performance on various components of the 


COMP Measurement battery and successful performance in work directly 
corresponding to those components. Regardless of the ultimate, success ' 


or failure df research like that underway in COMP, those engagedin the 
useof college admissions tests are well advised to continue to question 
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Finally, in the above discussion, the phr#se, ."predictive relationships 


>ve dT 
I fi th 


has frequently been usedjwithout specifying any measure or this relationship 

Ail too often, such relationships are reported, interpreted, and/or 
> . . * • 
dismissed 3olely in terms of some type of sample correlation coefficient'. 

To do''«o frequently obscures important facts, and possibly misleads 


"'‘tfftci sion^makers or others interested in interpreting a study ofNpredictive 
validity. In this regard, at lease three somewhat inuer-depeodent 
points can be made. First, frequently the nature^of the issues addressed 
by a predictive validity study suggest that a coefficient corrected for 
attenuation is m£re appropriate than a sample jcoefficient. Second, a 

I. ' . ■ 

correlation coefficient per se is seldom a sufficient basis for a practical 

deeisicn--at a minimum, one should also consider score variances and - - 

measures of error, such as the Standard error of estimate. Third, - / 
♦ ‘ • 

regression functions are almost always of more fundamental importance 
than the magnitudes o f sample ^correla tion coefficients! These points 
have been made by many writers, but they are sufficiently important td 
justify'repetition. 


Utility , and Prediction 


Predictions are properl/madfe only in conjunction with the various 


potential consequences of ygach pre' 


liotj^e 


e option. The concern ot-tne 


test developer and the .test user should not be -focused solely on"minimizing 


) •- > 



the qumber of cla ssific atioa errors, per se yi but also on the consequences 
of iiCfcscTassifications. ~ To' the point,' it T a r degirafel e li nk-questions . 1 
of prediction with concern, for utility or loss factions. 


Giveu t»fts Mice those 'contained is the ACT and other admissions 


. } .. . r •r ; v ' —: • . ’ * * * \ v . v-y- „ 


types ofv decisions 


. "'Vi' w'- 
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mad^dsiog data from these tests, (often in conjunction with other 
data), the particular loss functions pf interest vary widely, fn the 
case of the ACT testsfor exampLe, the consequences of classification 


errors could be markedly different when the tests are used in admission 
decisions by highly selective postsecondary institutions as opposed to 
open^admissions institutions,' th'e latter being far more frequent than^ 
the former at this time. 'In—tjie latter case, the uses'of the ACT tests 

with students typically include academic advising and placement. In 

. \ . " "*• • 

such uses, students are frequently able to~accept or ignpfe the counsel 
that the test data provide.- 

'• < ' / V’-: . ••• . * / - , . .•* i 

,*• -— . ’ * t «* ^ i 

Given the merits of the above observations^,a reasoned decision ©F 


a predictive nature requires, in principal., a join^_ consideration; of 

4 # f ^ ^ . . V. ; . 

both utilities, (or values) and statistical relationships of scores. It 
is necessary to acknowledge, however, that practical problems remain 


even if the user^-of test data (and/or'Other data frequently used in 




predicting'academic success) combines a concern for the expected conse¬ 


quences of decisions based on predictions with a parallel concern" for 

. • ‘ ' ' . ’ ^ ... v . 4 

the adequacy of the statistical procedures used to set the stage for 
those decisions. For example, college admissions testing programs &re. 

used in a vide variety-of contexts.for various different purposes; and. 

. - • , \ ■ 

Utility or. loss functions are .specific to both\:oatext and purpose, at 

least. However, ix is generally not\ simple matter for a particular 


decision-maker in^a particular college to specify and use ,losses or 
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O tfcrr Considerations 

•a 1 

v In chose instances where test and other data are used p r i ncipally 
■ •> , 
f or s election , de cisions (perhaps most frequently in the case of admissions 

testing for professional, schools), there are at\east two very important 

" -ij 

i . 

additional considerations. First, selection procedures based on grade 
predictions for all students may be unfair in some ways to certain 

groups (e.g., those of minority racial-ethnic background or of atypical 

Jt 

■ "■ 

age). When an institution is concerned only with maximizing the chances 
of. success of selected students, the use of a single pirebi^tive systems 
may be appropriate. However, when the institution's goals include 
concern for a kind of "fair" opportunity for admission for all subgroups - 
Vfltbin the applicant population, then other procedures may be required. 

, w • r *‘ «. ’ 

A second consideration^pout the'sole use of predi cted GPAs for 
selective, admissions isthe effects of that practice in limiting the 

I • •, - - - ’ . ' 

selected group to primarily academic achievers, to the exclusion of other 
types of ''achievers. -Even selective 1 institutions may wish to provide 

• ' . f- ■?. -o- , '* - 

opportunities for the slightly less academically taletted student with 
special talents in leadershipmusic, art, writing or other such.areas.’ 

suMrx • '■ n "; ' 

v .. • " v .. •* 

In conclusion^ the general goal of improving; predictions of academic 
success in^college is inherently worthwhile, as one aspects of a continuing 
effort to enhance a particular type of validity generally appropriate 
. for a college admissions .testyig program, That goal takes on practical . 

significance,howeveri only when it is considered in the context of the 

* V ■ • . v .-"*;&• . . 1 . ^ ■ *. . • •" - . • • •• . . ; . v „ . . 

various possible and likely uses of thejf redictions, the utilities 


mm 




t&Ose uses 4 to the ' 
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